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Abstract 

This paper presents the results of a pilot study investigating the use of the Realizeit adaptive learning 
platform to deliver a fully online General Psychology course across two semesters. Through mutual 
cooperation, UCF and vendor (CCKF) researchers examined students’ affective, behavioral, and cognitive 
reactions to the system. Student survey results indicated that students found the system easy to use and were 
generally positive about their seamless transition to adaptive learning. While the majority of students were 
successful, learning outcome metrics utilizing Realizeit indices indicated a potential for early prediction of 
students who are likely to be at risk in this environment. Recommendations are presented for the benefits 
of cooperative research between users and vendors. 


Background on Adaptiveness as a Learning Principle 

For the past several years, adaptive learning advocates have promoted its potential for transforming 
higher education. The concept, although considered by many a recent innovation, is anything but new. Even 
the most casual literature search will identify authors, professional organizations, industries, foundations, 
and government agencies advocating the importance of adaptiveness for learning (Bill and Melinda Gates 
Foundation, 2014; Levy, 2013; National DETA Research Center, 2014; Pacheco, 2014; Zmuda, Curtis, & 
Ullman, 2015). Supporting this position, many authors have valued adaptiveness as a functional concept. 
For example, Illich (1972) broached the issue in his chapter on learning webs where he pondered who could 
be served by bridges to nowhere, suggesting adaptiveness by using educational objects, skill exchanges, 
peer matching and professional masters. Certainly, these constructs have clear analogs in our technology- 
enhanced, educational culture. Johnson (2010) reinforced lllich’s (1972) learning web notion by describing 
the adjacent possible (that which is reasonably accomplished next) and liquid networks that feature 
feedback loops through idea exchange and refinement. Bates (1989) incorporated adaptiveness into her 
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berrypicking model for information search where a particular discovery opens multiple pathways to 
information sources. In her framework, uniquely individual approaches to information and learning are not 
only possible but highly likely. Hall (2004) argued that in actuality, adaptive learning geographies signify 
a reorienting process where students not only gain knowledge but improve their learning processes 
incrementally. Morville (2005) termed that concept wayfinding and Turchi (2009) suggested that the most 
appropriate framing metaphor for adaptiveness is mapmaking as a version of learning cartography. 

Other scholars approached the concept from even broader perspectives. Silver (2012) addressed the 
notion of progressive learning ability by using athletics, arguing that the path to significant achievement is 
rarely linear even for the most talented. According to him, rigidity impedes progress, while effective 
adaptive learning is best defined as controlled intensity. In a different framework, boyd (2014) argued that 
adaptability is critical to social networks because users must understand the interplay of context, audience, 
and identity in order to navigate virtual communities effectively—a seemingly important implication for 
learning networks. Shultz (2010) discussed the necessity of understanding that being wrong is a critical 
component in the learning process. She cited the writer Menand as saying, “...The right answer is, in a 
sense, a function of the mistakes” (p. 34)—suggesting that incorrectness increases the likelihood of 
following alternative paths. Levitt and Dubner (2014) were adamant in arguing that adaptiveness is the key 
to effective education, “The key to learning is feedback. It is nearly impossible to learning anything without 
it” (p. 34). Diamond (1997) further reinforced the adaptive feedback principle by characterizing the process 
as autocatalytic with continuous feedback loops, creating a sustaining synergy—an implicit reference to the 
go-at-your-own-pace capability of adaptive learning. Finally, Carpman and Grant (2012) provided a 
functional hierarchy for personal geographies (or adaptive learning) by what they identified as a directional 
sense. We augment their terms within the framework of learning: 

• Know where you are (baseline measures) 

• Know your destination (desired outcomes) 

• Follow the best route (learning path) 

• Recognize your destination (knowledge state) 

• Find your way back (repeat, revise and reassess) 

Adaptive learning is a critical component of many disciplines ranging from information search to 
forecasting and cultural evolution. However, no matter how diverse the disciplines are, there is apparent 
consensus about the core elements of adaptiveness: incremental learning, continual feedback, regular 
assessment, benchmarking, indexing growth and the availability of many paths to a final destination—the 
interaction of which alters the educational environment from a fixed setting to a flexible (adaptive) context. 

The Fundamental Premise of Adaptive Learning 

Like so many innovations in education and the social sciences, adaptive learning resonates with 
Senge and Roth’s (1999) mental model and Bowker’s and Star’s (2000) boundary object concepts. Both 
theories underscore the difficulty of forming a precise definition for the adaptive process. According to 
Senge and Roth, mental models are conceptual frameworks of generalizations by which we organize our 
thinking (1999). According to Bowker and Star, boundary objects have functional definitions in individual 
constituencies but become much weaker in the general community of practice (2000). Most of us 
understand that the fundamental objective of adaptive learning is to provide greater flexibility and multiple 
paths to achievement with reduced time constraints, but precisely how that happens for students is more 
difficult to specify. 

Adaptiveness in learning and assessment has been the topic of interest since the measurement 
pioneers such as Lord and Novick (2008), Rasch (1998), and Lazarsfeld (1995) began developing the theory 
and mathematics for adaptive testing, typically referred to as item response theory (IRT) (de Ayala, 2009; 
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DeMars, 2010). Their work demonstrated that item difficulty and individual ability levels can be equated 
and placed on the same logarithmically anchored scale, thereby freeing educators from classical 
measurement theory that requires examinees to respond to every item on the test. In practice, an 1RT 
examination presents an examinee with an initial item scaled with 1 to 3 parameters, and depending on 
whether or not they respond correctly, selects another item in progressing or decreasing difficulty. The 
process continues until there is assurance that the examinee’s knowledge has been accurately determined. 
Because items and ability have been equated, there is no requirement in 1RT that examinees be presented 
with exactly the same items or even an equivalent number of items. Currently, several national 
examinations, such as the nursing NCLEX certification (National Council of State Boards of Nursing 
[NCSBN], 2016), are administered using IRT, where one examinee may pass or fail with as few as 75 items 
presented, and another candidate may have to respond to 200 or more prompts before an accurate 
assessment of their knowledge level is achieved. Although this explanation is over-simplified, adaptiveness 
is well established and fully functional in the measurement domain—predominately apprising item 
characteristics. 

In a parallel development, Carroll (1963) addressed adaptiveness from a learning perspective. In 
doing so, he laid out the basic equation necessary for learning to be truly adaptive. His theory suggests the 
following: 

/time actually spent\ 

Degree of learning = f --—-- 

V time needed J 

The numerator in this formula is not elapsed time, but rather time actually spent on the learning process. 
The basic formula specifies that in order for a student to learn (or achieve a competency) he or she should 
have the time required to acquire mastery. This is the index for what has become an educational cliche. If 
how much time a student spends learning is a constant, then achievement will be the variable. If 
achievement is the constant, then how much time the student spends leanring is the variable. Carroll 
contended that the numerator of the equation is moderated by perseverance and aptitude. He defined 
perseverance as a function of willingness to: 

• Spend time beyond the ordinary, 

• Withstand discomfort, and 

• Tolerate failure. 

With respect to aptitude, he argued that the construct is best understood in terms of time needed to 
achieve mastery. Therefore, less time indicates higher aptitude and more time equates to lower aptitude. 
However, Carroll was quick to point out that aptitude is moderated by several other factors, the most 
important of which is prior learning—a fundamental starting benchmark for most platforms that support 
the adaptive learning process. Carroll claimed that his model was less related to learning theory, and much 
more relevant to the economics of learning. However, in the contemporary adaptive learning process both 
elements play an important role. In order to be adaptive, one must embrace a theory of how students learn, 
embedded in an optimally efficient process. 

The development of item response theory and Carroll’s model have made significant contributions 
to adaptive learning as we have come to understand it. The measurement community has demonstrated that 
adaptiveness is not only possible but functionally efficient and highly effective. Carroll showed us that we 
can conceive the learning process as one that is much more responsive to where the student is, where he/she 
needs to be, how he/she learns best, the most efficient path to get there, and the most effective method by 
which to assess his/her progress. 
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History of Adaptive Learning Research 

The advent of faster computer technologies coupled with the ability to process large datasets has 
fueled a newfound interest in developing adaptive learning systems. In addition, the strong push to improve 
student learning outcomes, including retention in higher education has created an urgency to harness the 
potential of adaptive systems to address these perpetual problems to personalize instruction. Table 1 
illustrates a variety of some (not all) adaptive learning systems developed at the time of this publication. 
Vendors and platforms provide a range of support and content. Content agnostic or open systems allow for 
more faculty or institutional control by providing the structure and logic of the adaptive shell but allowing 
for content to be developed by the institution—typically the faculty member, instructional designer or 
subject matter expert. Closed systems provide the advantage of off-the-shelf, previously developed courses 
that use the adaptive learning platform, but have minimal or no customization possible by faculty. These 
systems can provide a faster track to institutions utilizing adaptive learning because faculty are not 
encumbered by the significant time required to develop and personalize content. Often, these off-the shelf 
courses involve general education requirements or other offerings that may be similar in content across 
many higher education institutions (Brown, 2014; Tyton Partners, 2016). Perhaps in response to the 
additional time required to create or modify content, many vendors now provide instructional design 
support as well for those that provide significant authoring capabilities. 

Much of the recent literature on adaptive learning has been focused on examining current adaptive 
learning platforms and the promise of adaptive learning’s use for personalizing instruction (Brown, 2015; 
Tyton Partners, 2013, 2016). The definition of how truly adaptive a given software is varies widely, but 
generally the systems vary the path through the course material by student performance on assessments or 
student preference for a type of instruction. Approaches vary and adaptive algorithms are elusive for vendor 
products. Faculty may have limited ability to modify the adaptive course. 

While there is promise of instructional improvements as a result of adaptive learning, little research 
has been done in this area. Currently, a number of funding possibilities is creating the potential for 
investigation into the use of adaptive learning in higher education (Bill and Melinda Gates Foundation, 
2014; Association of Public & Land-Grant Universities, 2016; Online Learning Consortium, 2016). 
Restrictions on the variety of adaptive approaches and campus climates makes comparison and 
generalizability of findings difficult, however. This article focuses on the use of the Realizeit adaptive 
learning platform to administer an online section of General Psychology at a large, metropolitan university 
(for more information on Realizeit, please see http://www.realize.it/) . 

Adaptive Learning in the Realizeit System 

Realizeit incorporates probabilistic reasoning using Bayesian estimation procedures within an 
instructor-created learning network (Howlin & Lynch, 2014; Silver, 2012). The Bayesian procedure utilizes 
students’ initial baseline results to estimate their position in a curriculum framework. As learners progress 
through their adaptive courses, additional outcomes enable Realizeit to suggest alternative learning 
trajectories. This results in continuous updates of students’ ability estimates, the knowledge they have 
acquired, those objectives that still require mastery and recommendations for optimal paths through the 
course material. 

This design feature enables instructors to understand how learning components interact. Realizeit 
is capable of incorporating multiple learning resources into the curriculum such as video, audio, test, 
objective test questions, exercises and case studies, for example. 
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Table 1. Examples of adaptive learning systems 


Resource Name 

Content Source 

Acrobatiq 

Authoring platform with instructional design support with OER, faculty 
designed or off-the-shelf, pre-loaded, publisher or Acrobatiq content 

ALEKS 

Off-the shelf, not modifiable content 

Brightspace by D2L 

Authoring platform with instructional design support and content from OER, 
publishers or Brightspace Learning Object Repository 

Carnegie Learning 

Off-the-shelf courseware with minimal modifications possible 

Cerego 

Authoring platform and instructional design support with OER, real-time media, 
standards-aligned content, lesson plans, learner-generated materials 

CogBooks 

Authoring platform and instructional design support for off-the shelf courseware 
with content from OER, faculty, customers, publishers 

Difference Engine 

Authoring platform and instructional design service to customize course 
offerings from OER, publisher, proprietary client content 

Dri lister 

Authoring platform and instructional design service to customize course 
offerings including publisher content 

Fishtree 

Authoring platform where faculty can create content and modes, standards- 
aligned content, OER, real-time media, teacher lesson plans, premium publisher 
content 

Flat World 

Off-the-shelf courseware with pre-loaded, modifiable content including Flat 

World eBooks, OER, publisher content (VitalSource) 

Fulcrum Labs 
(formerly Adapt 
Courseware) 

Off-the-shelf, pre-loaded, customizable courseware from subject matter experts, 
peer reviewers 

Knewton 

Platform with content from partners with limited customization 

LeamSmart 

McGraw Hill off-the-shelf courseware generated by subject matter experts based 
on commercial texts. No customization by faculty 

LoudCloud Systems 

Authoring platform with customizable, standards-aligned, OER, real-time 
media, lesson plans, premium publisher content 

Lumen Waymaker 

Off-the-shelf with pre-existing OER premium BBC video content modifiable by 
faculty 

Open Learning 
Initiative 

Off-the-shelf courseware, preloaded with OER, pre-existing author materials, 
external, pre-developed learning activities 

Realizeit 

Authoring and delivery platform with instructional design services incorporating 
institution’s content, OER, publisher content and various media 

Sherpath 

Off-the-shelf, limited customizable content from Elsevier 

Smart Sparrow 

Platform and learning design service to design or import customizable content 
from various sources. 

Snapwiz 

Adaptive and collaborative learning platform importing customizable content 
from publishers, OER 
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Adaptability within the system bases itself on the following learning principles: 

• A student’s start position on an objective may be tailored by determining which concepts they have 
mastered. 

• A learner’s pathway through the curriculum—including revision and practice exercises—may be 
dynamically altered as they progress through the content based on learner behavior, attainment, 
performance, and progress. The system selects the most suitable content for students as they 
undertake a course module, given their learning requirement at that time. 

• The system selects the most appropriate pedagogical elements (learning bits) within a concept or 
objective. 

• Adaptations to learning paths are based on rules specified by the instructors or selections made by 
the students themselves. 

Within Realized, the main source of adaptability originates from the intelligence engine discussed 
above; however, the instructor or the students may themselves personalize course content. Realizeit 
integrates with Learning Management Systems (LMS) such as Instructure Canvas to provide platform- 
based learning navigation. As students begin a class assignment in the LMS they are launched seamlessly 
into the equivalent Realizeit objective. The advantages of this process include single sign-on authentication, 
course and group synchronization, and automatic grade transfer to the LMS gradebook, including additional 
metrics and comments. 

At almost every point in the learning process, the student has final control over learning and next 
steps within the system. They may alter their learning path progression (trying new concepts) and 
alternatively undertake review (revising/practicing previous concepts) procedures. In addition, they are able 
to view supplemental learning material including adding, removing, and reordering course elements within 
the content. However, this is not a completely open landscape for students but is structured for optimal 
learning while allowing for flexibility. Guidance is provided by the Realizeit system which directs students 
towards ability-appropriate activities to increase the potential for success. Additionally, instructors have the 
ability to modify learning objectives for students within their courses. This process can be facilitated with 
analytics data that improves interaction and intervention with students, ranging from asynchronous 
messages to real-time intervention. 

General Psychology in the Realizeit Adaptive Platform 

General Psychology is an introductory course, offered as one of the social foundations options 
required as part of the General Education Program (GEP) for undergraduates and also as part of the 
Psychology undergraduate curriculum. Because Psychology is the largest undergraduate major, resulting in 
substantial enrollment for the General Psychology course, the department chair (Dr. Jeff Cassisi) 
determined that adaptive learning might be a good fit for helping increase access, and improving learning 
outcomes for students choosing this class as part of their undergraduate program. 

One section of PSY2012: General Psychology was redesigned by Dr. Jeff Cassisi to utilize the 
Realizeit adaptive learning platform. Online courses at UCF are delivered through the Instructure Canvas 
LMS, which UCF has branded Webcourses. Within Webcourses, UCF online courses typically follow a 
similar format, giving students a class “front door” that provides access to course announcements, 
gradebook, modules, discussions, syllabus, and any tools the instructor may utilize within the course. 
Webcourses email allows students to connect with instructor and students within a given course. All of this 
structure remained constant for the adaptive learning online course, so that students who were familiar with 
online courses at UCF would be familiar with the structure of this course as well. 
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Psychology course content was divided into eight modules (history & research, biology & 
consciousness, human diversity & development, sensation & perception and learning, memory and thinking 
& language, intelligence & motivation, emotions & health and personality, social psychology & 
psychological disorders), each corresponding to approximately two chapters in the required course 
textbook. The link to each of these modules opened the Realized interface for students to engage with the 
adaptive content. When students completed a module, their assessment scores were then passed back to the 
Webcourses gradebook as one of the course module grades. 

Overall course grades were comprised of the student grade in each of the eight modules (10% each; 
80% of total grade), an immersive research experience required of all General Psychology students (10%), 
and a comprehensive, standardized (non-adaptive) final exam (10%) that is administered in each General 
Psychology course. Criteria were set by the instructor within Realized and used by the system to calculate 
the grade passed back to the Webcourses gradebook for each module. Within the syllabus, students were 
given a schedule of suggested due dates to use as a guideline for managing their time in order to complete 
all modules by the end of the given 16-week semester session and prior to the final exam due date. Students 
were free to review the content within a given module to revise and improve their grade as many times as 
desired throughout the semester. Access to the modules closed the day before the last day of the final exam 
which was open to students for a one-week period. 

The Study: An Evolving Research Partnership 

This study assessed adaptive learning in General Psychology by assessing students’ affective, 
behavioral, and cognitive involvement within Realizeit. Initially, the study evaluated student reaction to 
and satisfaction with learning in this modality, concentrating on positive and negative elements plus features 
that created ambivalence. Secondly, data were collected about how students interacted with the content, the 
instructor, and each other when they were free to traverse the course at their own pace. Subsequently, the 
investigators compared the performance of successful and unsuccessful students (gauged by a grade of C 
or better) on each of the eight course modules. These components of the research project were completed 
by the Research Initiative for Teaching Effectiveness (RITE) at the University of Central Florida (UCF) 
working with Dr. Jeff Cassisi, the instructor of record for the course. 

In the contemporary educational environment the historic distinction between universities and 
vendors is beginning to blur just as the boundaries among disciplines and even colleges are much less 
rigidly defined. Certainly, both UCF and CCKF, the developer of Realizeit, have vested interests in adaptive 
learning but they have come to understand that cooperative research ventures strengthen both organizations. 
Further, both entities realize (no pun intended) that shared research and data advance both parties’ 
understanding of how to implement adaptive learning in the most effective manner. The fundamental 
partnership premise for this study was that UCF and CCKF enter into a working agreement where data are 
shared, cross-analyzed, validated, presented, and published in an objective manner evaluating the adaptive 
learning process in the many contexts in which it is likely to appear. This study represents the first product 
of that partnership among UCF, the instructor, and CCKF—a new research protocol. In fact, this paper is a 
call for other vendors and universities to do the same so that the collaboration might be expanded. For us, 
this is valuable because UCF can do things CCKF cannot and CCKF can do things that UCF cannot. We 
need each other in order to progress most effectively and improve the research paradigm. 

The Data Collection Protocol 

Students’ reaction to their adaptive learning experience was obtained through a questionnaire that 
was developed and validated by students, the instructor, and investigators from RITE. The initial version 
of the instrument was developed from a review of research on adaptive learning in higher education. From 
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this review the investigators drafted a preliminary data collection protocol that captured student 
demographic information and addressed their perceptions with issues such as: 

• Learning effectiveness 

• Engagement and re-engagement with adaptive learning 

• Learning personalization 

• Seamlessness of the Realized platform 

• Learning assessment accuracy 

• Sequence effectiveness 

• Guidance provided by the system 

• Platform responsiveness 

• General satisfaction with the system 

The initial version of the instrument was revised according to the recommendations of faculty and 
CCKF researchers and pilot-tested with a group of students who had experience with learning in an adaptive 
system. Based on faculty and student recommendations, the final items were crafted and structured in a 
five-point Likert response format augmented with open-ended questions for more granular detail from 
students. 

Achievement in each of the eight course modules was based on an assessment protocol developed 
by the instructor within Realizeit, producing a final score on each module for each student. These scores 
permitted comparison of the achievement levels of successful and unsuccessful students as the course 
progressed, documenting learning trends of those two groups. 

In addition to outcome measures created by the instructor, the Realizeit system generates a large 
number of student performance analytics, internal to the system. Two of these indices were subjected to 
further analysis. The first outcome measure used in the study was an index created by Realizeit termed 
knowledge state. This denotes the average ability that students achieve in a module for which there is direct 
evidence (some assessment exercise) across all the nodes. For example, if there were 10 learning nodes for 
a particular module and a student achieved the maximum possible assessment score on each of them, their 
knowledge state score woidd be 100%. Knowledge state in the Realizeit adaptive learning system is 
expressed as a ratio of achieved to possible accomplishment. The second was percentage of nodes covered 
by each student for the course modules. For instance, the module human diversity and development is 
comprised of a number of learning nodes that may be typified by readings, videos, quizzes, activities, case 
studies, assignments, postings, discussions, etc. If students complete all activities associated with a module 
within Realizeit, the system records 100% for percentage of nodes covered, thereby representing a surrogate 
for the degree to which students engaged in this module (and throughout the course). 

The Study Sample 

The sample for this study was comprised of students who registered for the fully online adaptive 
learning section of General Psychology in the fall 2014 or spring 2015 semester (n=292). Students self- 
selected the fully online course, but were not aware of the use of adaptive learning until the first week when 
the class opened. The syllabus described the use of the adaptive learning system, Realizeit, as well as 
grading procedures. Students provided responses to the questionnaire and were tracked for achievement in 
the course through instructor-developed assessment and engagement with Realizeit through system 
analytics. Student demographic survey responses indicated that on average they were 19 years old, 57% 
female and 42% male, with academic class breakdown of freshmen (33%), sophomores (29%) juniors 
(21%), and seniors (13%). Thirty-one percent of respondents classified themselves as white/Caucasian, 


Online Learning - Volume 20 Issue 3 - September 2016 


81 



18% Hispanic/Latino, 12% Black/African American, 6% Asian, and 30% preferred not to disclose their 
ethnicity. 


Methodology 

Before assessment of the student responses to the questionnaire was completed, the psychometric 
quality (domain sampling) of the information yielded by the instrument was determined. The reliability was 
computed using coefficient alpha (Cronbach, 1951). Frequencies for each of the survey items were 
calculated and examined along with means and standard deviations for those who engaged with various 
features of Realizeit. 

Kaiser and Rice (1974) developed a measure of sample adequacy (MSA) that provides evidence 
about whether or not the student responses produce an adequate psychometric sample that is a set of items 
that properly represents a domain of interest. The index is limited by 0 and 1 and if the value is in the .80 
to .99 range, the investigator has evidence of a proper domain sample. Values in the .70s signal a generally 
acceptable result, and those in the .60s indicate data that are unacceptable. MSA has been used as a data 
assessment index before any factoring analyses are completed. This procedure is recommended by Dziuban 
and Shirkey (1974) prior to any latent dimension analysis and was used with the data obtained for this study. 

These data provided baseline information for determining what degree of favorableness toward this 
learning mode might be expected. In addition, these responses were the identifying sources of potential 
problems and steps that need to be taken in order to improve the adaptive learning environment and support 
for students. 

Dimensionality of Student Responses 

The investigators sought to determine the underlying dimensionality of students’ perceptions about 
their adaptive learning experience. This can be accomplished through the application of some variant of the 
factor analysis procedure. In this study the method used was maximum likelihood factor analysis (Muliak, 
2009). This approaches has the advantage of producing pattern coefficients that are maximum likelihood 
estimators of population values. In addition to traditional methods for determining the number of common 
factors to extract, the procedure uses a goodness-of-fit statistical test as well. 

The number of factors retained in the final solution was accomplished by using three procedures. 
The first two, goodness of fit and eigenvalues of the correlation matrix greater than one appear extensively 
in scientific literature (Mulaik, 2009). However, a third technique originally proposed by Dziuban and 
Shirkey (1993) and validated by Hill (2011) was incorporated into the process as well. The algorithm 
initiates with an initial assessment of the data set with the MSA followed by iterative MSA computation on 
the matrix of partial correlations remaining after each factor is removed. Once a value in the .60s is 
encountered, there are no more dimensions to be derived from the data. What remains is basically noise. 
Kaiser and Rice (1974) demonstrated that in addition to the overall MSA for a correlation matrix, individual 
MSA values can be computed for each item on the questionnaire. This provided the investigators with a 
final opportunity to screen their data and improve the domain sampling. The individual MSAs were 
computed for each item and those with values in the .60 range were removed—because in Kaiser’s words 
they don’t belong to the family psychometrically. This resulted in an increase in the overall MSA from .81 
to .89. The initial pattern matrix was transformed (rotated) according to the promax procedure (Hendrickson 
& White, 1964). Pattern coefficients absolutely larger than .30 were used for interpretation purposes. 

Once the final dimensionality was completed, factor scores for each subject in the sample were 
derived using the Anderson and Rubin (1956) method. These scores have a mean of zero and a standard 
deviation of one and produce a reasonably good relationship to the estimated factor validity. The scores 
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were then arbitrarily rescaled to have a mean of 50 and standard deviation of 10 for ease of interpretation 
(T-score transformation). Subsequently, the investigators used the question “Realized helped me learn” 
with the K-Means clustering procedure (Hartigan & Wong, 1979) to identify noteworthy sub-groups in the 
sample that might respond differently at the latent trait level. 

Module score means were tested for significant differences and effect sizes computed for the 
successful and unsuccessful students (Hedges, 1981). In addition, trend lines on the module means and 
standard deviations for those groups were constructed. Finally, the means and standard deviations for 
knowledge state and knowledge covered were compared for the top 20 and bottom 20 performing students 
and the class as a whole. 


Results 

Student attitudes toward adaptive learning and the Realizeit platform 

Overall, students were positive about their experience with adaptive learning in the Realizeit 
platform. Table 2 illustrates the means (5=high; l=low) and standard deviations for each question of the 
survey. Survey questions were selected to assess the usability of the Realizeit system—whether students 
found the system and its various functions and features easy to use, how well students felt the system 
captured their knowledge through the difficulty of items and pathways presented to them, and the student 
perceptions of Realizeit’s ability to help them learn and remain engaged. In addition to demographics, 
students were asked their overall satisfaction with the system and its components—whether it became 
personalized to them, if they followed system recommendations, their perception of time spent in the 
system, and finally, whether they would take another class using Realizeit if given a choice. 

In terms of system usability, the majority of students felt the system was easy to use, as were the 
learning path and guidance panel features, with clear instructions provided. While over half of the students 
indicated they felt the guidance panel was helpful and easy to use, 23% indicated they did not use this 
feature or did not know what it was. Seventy-seven percent of the class indicated that they felt Realizeit 
provided them with the feedback necessary to stay on track. 

Students (74%) believed the system accurately assessed their ability levels in psychology and that 
the grades accurately reflected their ability. Three-fourths of the class felt that the system increased their 
engagement with the course content over traditional instructional methods. 

In rating the difficulty levels, the majority of students felt the difficulty levels of the learning path 
sequence (45%), material (46%), and questions asked (47%) were neither easy nor hard. Few found each 
to be somewhat or too difficult (13%; 16%; 28%, respectively). 

The system provides students with suggested “next steps” although students can choose to ignore 
that suggestion. Survey respondents indicated that over half of students (53%) always or quite often 
followed the system recommendation while 28% sometimes did and 18% rarely or never followed these 
system suggestions. Nearly half of the students (47%) felt they spent more time in Realizeit compared to 
other formats, while 54% indicated they felt they spent an equivalent amount of time or less than in a non- 
adaptive format. Many of the students (64%) reported that the system became personalized to them over 
time. And, 78% of students indicated that they woidd take another course using Realizeit if given a choice. 

As is often the experience when utilizing a platform (or any instructional technique) for the first 
time, there were minor system issues that were resolved as the semester progressed. As such, the 
investigators queried students about whether these “glitches” impacted them in any appreciable manner. 
However, 92% of those surveyed indicated that they did not experience any technical issues with the system. 
Only 1% of students (n=3) had issues they felt were not adequately resolved. 
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Table 2. Student reactions to survey items 


Item 

n* 

X 

SD 

Realized helped me learn the course material 

241 

4.02 

.92 

Realized’s assessment exercises were effective 

235 

3.82 

.78 

Difficulty of the “learning path” sequence 

240 

3.35 

.81 

Difficulty of the learning material 

241 

3.24 

.78 

Difficulty of the questions asked 

239 

2.99 

.80 

Realizeit increased my engagement 

233 

3.92 

.87 

Grading accurately reflected my knowledge 

229 

3.81 

.86 

Ability levels reported by Realizeit were accurate 

235 

3.79 

.84 

I would take another course using Realizeit 

234 

4.09 

.99 

Realizeit system became personalized to me 

228 

3.67 

.87 

I followed recommended “next steps” 

239 

3.51 

1.11 

Time spent in Realizeit 

229 

3.31 

1.15 

Realizeit was easy for me to use 

234 

4.24 

.77 

The instructions in Realizeit were clear 

241 

4.12 

.80 

“Learning Path” was easy to use 

184 

4.01 

1.00 

“Guidance panel” was easy to use 

211 

3.91 

.89 

Realizeit provided me with the necessary feedback 

237 

3.86 

.78 

“Guidance panel” was helpful 

182 

3.81 

.67 


*differing n’s represent missing data 


Factor Results 

The promax transformed pattern matrix for the maximum likelihood analysis of student responses 
to the Likert scale items regarding the Realized adaptive platform is presented in Table 3. All three methods 
for determining the number of common factors converged on three dimensions. The goodness of fit 
probability was .26 and three eigenvalues were greater than one. The residual MSA after extracting three 
factors was .54. Individual MSAs ranged from .70 to .92. The alpha reliability coefficient was .89 and the 
variables exhibited virtually no complexity indicating an acceptable solution. 
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Table 3. Promax Transformed Pattern Matrix — Maximum Likelihood Factor Analysis (n=244) 


Item 

Component 

MSA 

1 

2 

3 

Realizeit helped me learn the course material 

.72 

-.24 

.03 

.71 

1 would take another course using Realizeit 

.71 

-.06 

-.03 

.73 

Ability levels reported by Realizeit were accurate 

.68 

.05 

-.11 

.70 

Realizeit increased my engagement 

.68 

-.06 

.00 

.76 

Realizeit provided me with the necessary feedback 

.65 

.08 

.08 

.77 

The instructions in Realizeit were clear 

.64 

.03 

.02 

.92 

Realizeit’s assessment exercises were effective 

.59 

.21 

-.09 

.89 

The grading accurately reflected my knowledge 

.52 

.23 

-.11 

.92 

The Realizeit system became personalized to me 

.51 

.08 

.07 

.91 

Realizeit was easy for me to use 

.50 

-.01 

.22 

.87 

The difficulty of the learning material 

.04 

.71 

-.01 

.89 

The difficulty of the sequence 

-.01 

.56 

.16 

.89 

The difficulty of the questions asked 

-.01 

.52 

-.03 

.89 

“Guidance panel” effective 

.00 

.03 

.63 

.89 

“Learning Path” effective 

-.01 

.05 

.56 

.90 

Eigenvalues 

4.7 

1.9 

1.3 


X 2 = 69.7 p = .26 





Factor Correlations 

.15 





.39 

.19 



Overall MSA = .86 






Residual MSA = .54 
Alpha = .89 
Components: 

1 = adaptive effectiveness 

2 = course noise 

3 = Realized guidance 

The first factor named adaptive effectiveness indicated that students in this environment evaluate 
their course with a dimension that reflects their judgment about whether adaptive learning represents a 
viable course modality. By evaluating the Realized platform, students responded to adaptive learning by 
how well it facilitates their learning, engages them, assesses their progress and clearly specifies the rules of 
engagement. The second factor represented a dimension named course noise that relates to the degree to 
which students encounter interference in the learning process emanating from the course model and 
underlying technology. In a sense this is the did adaptive learning get in the way factor. The final dimension, 
Realized guidance, indexes the core value of adaptive learning because students judged Realizeit by how 
effectively the system’s guidance algorithms helped them learn the material and acquire the concepts of 
general psychology. The factor correlations range from .39 to .15 indicating a general independence with 
the strongest relationship found between effectiveness and guidance. 

Factor Score Analysis 

The means, standard deviations, significance levels and effect sizes on the rescaled factor scores 
for the two student clusters based on the “ Realizeit helped me learn ” item are presented in Table 4. 
Graphical portrayal of that relationship is presented in Figure 1. Based on those results the two groups 
responded differently about adaptiveness being an effective learning climate (p=.00, ES=2.28). Cluster one 
appears strongly positive while cluster two is somewhat more ambivalent toward this approach. The number 
of students in the positive group was greater than three times the number in the ambivalent group, indicating 
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that the large majority viewed adaptive learning as an effective learning device. The ambivalent group, 
however, saw significantly more course noise (p=.01, ES=.38) and found the guidance feature of Realizeit 
significantly less helpful (p=.03, ES=.36). From these results there appears to be an interaction among 
factors about how students respond to adaptive learning. The variability for the effective climate factor 
scores for the two groups was significantly different for the groups (p=.00). The groups were not significant 
on variability for noise and guidance (p=. 14 and p=.82). 


Table 4. Comparison of cluster factor scores for adaptive effectiveness, course noise, and Realizeit 
guidance. 




n 

X 

SD 

P 

HVP 

ADJP 

ES 

Adaptive effectiveness 

Cluster 1 

188 

52.9 

7.5 

.00 

.00 

.00 

2.28 

Cluster 2 

54 

39.8 

10.8 

Course noise 

Cluster 1 

188 

49.1 

9.4 

.01 

.14 

.03 

.38 

Cluster 2 

54 

52.9 

11.5 





Realizeit guidance 

Cluster 1 

188 

50.7 

9.9 

.03 

.82 

.03 

.36 

Cluster 2 

54 

47.2 

10.2 


Cluster 1 = positive 
Cluster 2 = ambivalent 
HVP = variance homogeneity 
ADJP = Welch adjusted probability 
ES = Fledges’ g 


60 

55 

50 

45 

40 

35 

30 



Adaptive Course Noise Realizeit Guidance 

Effectiveness 


Figure 1. Factor score means comparison based on K-means cluster of Realizeit helped me learn 


Module Scores for Successful and Unsuccessful Students 

Table 5 presents some noteworthy results showing that after the first module in the class (Flistory 
& Research) there was a rapid divergence of the success and nonsuccess groups’ achievement. This module 
showed a difference of 9.3 compared to 9.0 (p=.05, ES=.43). In addition, for Module 1 the variances in the 
groups showed no significant difference (p=.17). This module covered the basic “introductory” 
information. However, after that module, the drop-off for the non-successful group was pronounced and 
rapid. All mean differences were significant, as were differences in the variability of the groups. For Module 
8 (Social Psychology & Psychological Disorders) the mean differences were 9.1 compared to 1.2 (p=.00) 
with a large effect size (ES=5.13). In fact, the difference in performance on Module 2 (Biology & 
Consciousness) for the two groups (9.5 compared to 7.4, p=.00) is an effective predictor for who will 
succeed and who will not if no further intervention is initiated. These findings suggest that there may be an 
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analytics component embedded in adaptive learning. Figures 2 and 3 portray the important findings 
graphically. Figure 2 demonstrates the early and rapid achievement decline of non-successful students. 
Figure 3 demonstrates an equally noteworthy finding. After Module 1, the variability in performance of 
non-successful students is large compared to those who succeed. Non-successful students are considerably 
different from each other in the way they approach learning. 


Table 5. Comparison across Realizeit module scores for successful and unsuccessful students 


Module 


n 

X 

SD 

P 

HVP 

ADJP 

ES 

1: History & Research 

Success 

Non-success 

278 

14 

9.3 

9.0 

.67 

.94 

.05 

.20 

.17 

0.43 

2: Biology & Consciousness 

Success 

Non-success 

278 

14 

9.5 

7.4 

.40 

3.3 

.00 

.00 

.00 

2.42 

3: Human Diversity & 

Success 

278 

9.4 

.61 

.00 

.00 

.00 

2.61 

Development 

Non-success 

14 

6.5 

4.7 





4: Sensation & Perception 

Success 

278 

9.4 

.42 

.00 

.00 

.00 

3.04 

and Learning 

Non-success 

14 

6.1 

4.2 





5: Memory and Thinking & 

Success 

278 

9.4 

.69 

.00 

.00 

.00 

3.47 

Language 

Non-success 

14 

5.4 

4.4 





6: Intelligence and 

Success 

278 

9.4 

.75 

.00 

.00 

.00 

4.20 

Motivation 

Non-success 

14 

4.5 

4.1 





7: Emotion & Health and 

Success 

278 

9.3 

.53 

.00 

.00 

.00 

6.41 

Personality 

Non-success 

14 

2.8 

3.4 





8: Social Psychology & 

Success 

278 

9.1 

1.4 

.00 

.00 

.00 

5.13 

Psychological Disorders 

Non-success 

14 

1.2 

3.3 






HVP = variance homogeneity 
ADJP = Welch adjusted significance 
ES = Hedges’ g 



Figure 2. Module mean scores for successful and unsuccessful students 
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Figure 3. Module standard deviations for successful and unsuccessful students 

CCKF Analysis 

All results presented to this point were generated by RITE working with the course instructor. 
However, in the study section of this article the authors indicated that this research project was cooperative 
venture between RITE, the psychology instructor and the research unit of CCKF, the developers of Realizeit 
(chief research scientist, Colm Howlin). CCKF identified the top and bottom 20 students in the classes 
across all achievement measures in the platform and compared them with the class as a whole on the two 
measures (knowledge state and knowledge covered) for each of the eight modules. Clearly, there are 
dependences in these data since the top and bottom 20 students are included in the overall class data, thereby 
precluding any meaningful significance testing that requires independence. However, from a descriptive 
perspective Table 6 and Figures 4 and 5 present useful information. Table 6 shows that the percentage 
scores for the top and bottom students are 95.6 and 58.8, respectively while the average for the class is 89.4. 
This shows that there is an approximate 37% achievement differential between the top and bottom groups, 
a 31 % difference between the class and the bottom group, with only a 6% difference between the top twenty 
students and the class in general. This is an interesting finding discovered by CCKF. Figure 4 demonstrates 
that the knowledge state profiles for the top 20 students and the class are approximately the same shape, 
close to each other, and virtually at the same level. However, the bottom twenty show marked dissimilarity 
on all three of those profile characteristics, suggesting that there are distinctly different groups in the 
adaptive learning environment. In addition, there is a large standard deviation in the lower group compared 
to the top twenty and the class (18.1 compared to 1.2 and 1.6, respectively). 

Table 6. Comparison of knowledge state percentage and knowledge covered percentage across 


modules for the top and bottom 20 students with the class as a whole * 




X 

SD 


Top 20 

95.6 

1.2 

Knowledge state 

Class 

89.4 

1.6 


Bottom 20 

58.8 

18.1 


Top 20 

99.9 

.07 

Knowledge covered 

Class 

94.7 

2.4 


Bottom 20 

58.8 

21.4 


* Completed by Colm Howlin, chief research scientist, CCKF 
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Figure 4. Knowledge state percentage across modules for the top and bottom 20 students compared to the 
entire class. (Completed by Colm Howlin, chief research scientist, CCKF). 
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Figure 5. Knowledge covered percentage across modules top and bottom 20 compared to the entire class. 
(Completed by Colm Howlin, chief research scientist, CCKF). 

Table 6 for knowledge covered shows a very similar trend. This variable indicates the percentage 
of nodes completed across the eight modules. One may observe from the table that the top twenty students 
completed virtually all of the modules (99.9%). Simply stated, they did all the work. The class in general 
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was not far behind completing 94.7% of the nodes for the course. The bottom group, however, completed 
approximately 59% of the work available in the course—40% less than the top group and 36% less than the 
class in general. The standard deviation for knowledge covered shows a pattern similar to knowledge state. 
The top 20 and the class are close (.07 and 2.4), although to some extent this is an artifact since there is 
virtually no variability in the top group. The bottom twenty students exhibit much more variance in their 
behavior (SD=21.4). Figure 5 demonstrates a profile pattern for knowledge covered very similar to the one 
for knowledge state (Figure 4). The top 20 students and the overall class are very similar while the bottom 
group demonstrates noteworthy and a different course engagement pattern. Both of these findings created 
by CCKF reinforce those found by RITE for the module scores. Early in the course, the adaptive learning 
environment offers the potential for identifying future at-risk students by considering their engagement 
patterns. 

Limitations 

There are several limitations associated with this study that impact the methods, data analysis, 
results, and conclusions. Most important is that although hypothesis tests were completed throughout the 
data analysis, the findings should be considered primarily descriptive. Because the students self-selected 
into the classes, randomization was not possible and many of the assumptions associated with these tests 
were not accommodated. Therefore, the probability levels reported should be interpreted with caution, 
although the authors contend that the procedures used for data analysis are relatively robust with the 
exception of random assignment. 

A second limitation arises from the fact that although this study was conducted over two semesters 
it lacks a longitudinal component. This study failed to address the overriding long-term impact of adaptive 
learning in the study of psychology. In addition, this work took place in an introductory class and did not 
address the implications for more advanced topics that involve hierarchical structures, such as 
industrial/organizational psychology, statistics, or clinical courses, for instance. Although the eight modules 
of the course have an imposed sequence, they are relatively independent of each other. Therefore, they may 
not capitalize on the topic prerequisite dependencies assumed in adaptive platforms. 

In addition, the self-report aspect of student responses to this learning modality leave some 
unanswered validity questions. Traditional Likert scales force students into approximate categories that 
ignore the within-group variability. There was no way to address that issue in this study so that the ratio of 
between to within variance was ignored by using measures of central tendency or reporting individual 
category responses. Although great care was taken to establish proper domain sampling, the factor analytic 
work here was completed on individual items with their presumed low reliability. Once again this limits 
the generalizability of these findings. 

Finally, the pedagogy and technology involved in adaptive learning is continually changing. The 
interplay between the instructor and the developers of Realizeit resulted in platform and course 
modifications throughout the semester, creating slightly nuanced versions of adaptive learning experienced 
by the students. At the moment this modality is in a continual state of flux so that although this study may 
appear to be focused, baseline work is more directly a boundary obj ect relating to the complexity of adaptive 
learning. 

Discussion 

Adaptive learning presents opportunities and creates challenges for higher education. The 
fundamental nature of these adaptive systems allows students to immerse themselves in content and 
progress through the material at their own pace-in theory, as quickly or as slowly as an instructor-defined 
course structure allows. Flowever, this creates the possibility of an isolated environment for students who 
interact primarily with the system and infrequently with each other. In examining this question, the survey 
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results indicated that students perceived themselves as interacting less or much less (75%) with each other 
than in a course not using Realized (Table 7). However, the researchers wondered how much students 
wanted to interact with each other in the classroom. Only 20% of them preferred some or substantial 
interaction with other students; 29% had no preference; and 52% of those responding indicated that they 
preferred little or no interaction with other students in their classes (Table 8). When indicating their 
preferred method of interacting, the majority of students (59%) indicated discussion boards-possibly 
because this is the predominate method used within the online instructional environment and therefore the 
one with which they are most familiar (Table 9). 


Table 7. Frequency> of perceived interaction with other students compared to non-adaptive class. 


Response 

% 

Much less 

51.3 

Less 

23.9 

About the same 

18.5 

More 

4.2 

Much more 

2.1 


Table 8. Preferred frequency of interaction of students in class. 


Response 

% 

None 

22.1 

A little 

29.6 

No preference 

28.7 

Some 

17.1 

A lot 

2.5 


Table 9. Preferred method of interaction for students. 

Method 

% 

Discussion boards 

59.4 

Live chat sessions 

16.0 

Live whiteboard 

17.6 

Virtual group conferences 

8.6 

Other 

15.6 


This isolation effect is not new to adaptive systems, and in fact, is an issue in many fully online 
courses where effective pedagogical course design becomes a critical element. While students learn the 
content through immersion in the Realizeit platform, features of the learning management system can be 
utilized to provide additional resources for faculty to develop course scaffolding to create the most effective 
and interactive online environment to support student learning. However, given the results of this study 
where success rates were remarkably high, combined with the independent nature of the course modules 
and the knowledge-comprehension level of this course material, the learning benefit of interaction might 
come into question. This does seem at odds with current digital learning theory. However, knowledge 
acquisition in a go-at-your-own-pace environment may minimize the necessity of peer-to-peer interaction. 
Further research on these issues is necessary, especially for courses that require skills such as evaluation, 
analysis, synthesis, group problem solving, discovery learning and critical thinking. 

With a system such as Realizeit, objective and node granularity is a critical component of the course 
design. Adaptive learning is driven by a decision engine that requires prior information in order to determine 
an optimal learning path for students. In order to do this effectively it needs adequate metaphoric degrees 
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of freedom in order to “have room to operate.” Modules with too few associated nodes can paralyze the 
system with respect to making accurate decisions. Too many nodes can overburden students with a daunting 
set of tasks. Although design is a critical component of any effective course, this becomes particularly 
important with true adaptive leaning, particularly those open, content-agnostic systems which allow faculty 
control over course design. The high front-end load in creating a course can pay immeasurable dividends 
in the student learning cycle. Adaptive learning demands the Goldilocks instructional design principle. The 
porridge cannot be too hot or too cold—it must just right. Adaptive courses cannot be too coarse or too 
granular. Classes must be designed at level of indenture that allows students to effectively manage their 
own learning with the support of a system. 

Adaptive learning resonates with educational theories contending that students must be allocated 
the time needed to master facts, concepts or constructs. Additionally, it responds to the well understood 
principle that prior learning is necessary for the attainment of successively higher levels of achievement. 
Incorporating these two principles into learning design, although seemingly straightforward, represents a 
seismic shift in the traditional educational arrangement. Currently, courses are designed by incorporating 
certain constraints, even in the online environment. Semesters are sixteen weeks long. The final 
examination is administered during a specified time or time period. Assignments have due dates. The course 
ends. Grades are due by a specified date. These constraints are the means by which we manage the 
educational enterprise for students and instructors at the present time in higher education. However, most 
of these requirements can be loosened or eliminated in the adaptive learning environment. Theoretically, 
students can move at their own pace and are assessed for competence when they or the system deems they 
are ready. They can finish a course early or they can continue working after the semester ends. The 
implications of this thinking are substantial. Currently, if a student finishes a course early they might have 
to wait to take the exam at the end of the semester. If we are to embrace truly adaptive learning and gauge 
a true index of its effectiveness much of the current educational structure will require reconsideration. 

Steven Johnson (2004) documents the influence of six phenomena in the world culture: clean, time, 
glass, light, cold and sound. The compelling aspects of these developments is that over decades and 
centuries they caused a profound change in the lives we lead today, but no one ever saw it coming. Forrester 
(1991) makes the case that: 

1. We just cannot anticipate how an intervention will ripple through a complex system 

2. Many outcomes will be counterintuitive 

3. There will be positive and negative unanticipated side effects. 

We speculate that should we stay the course with adaptive learning, we will encounter much of 
what Johnson and Forrester have demonstrated. The changes in our educational system will be profound 
but these changes, although potentially beneficial at many levels, may cause considerable angst and 
discomfort. In spite of that, adaptive learning can find its proper place in our transformed educational 
system. Johnson (2006) puts it this way: “History has its epic thresholds where the world is transformed in 
a matter of minutes—a leader is assassinated, a volcano erupts, a constitution is ratified. But there are other, 
smaller, turning points that are no less important. A hundred disparate historical trends converge on a single, 
modest act—some unknown person unscrews the handle of a pump on a street in a bustling city—and in 
the years and decades that follow, a thousand changes ripple out from that simple act. It’s not that the world 
is changed instantly; the change itself takes many years to become visible. But the change is no less 
momentous for is quiet evolution” (162). 


Conclusions 

This small pilot study intended to answer fundamental questions about the impact of adaptive 
learning in higher education. Because this modality deviates considerably from traditional formats 
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including face-to-face, online, blended, and lecture capture, understanding the way it impacts students, 
faculty members, and course structure becomes fundamentally important. The investigators sought to 
determine how students function in a learning environment where they experienced greatly increased 
learning latitude. Further, it was deemed important to discover what difficulties would be encountered as 
well what modifications might be necessary for this approach to be effective. In addition to gauging student 
responses, the study focused on how students engage and achieve, and what important subgroups might be 
identified from a predictive analytics perspective. How early can struggling students be identified? What 
outcome measures will be most informative for adaptive learning courses and how are traditionally accepted 
pedagogical assumptions challenged? However, as with many pilot studies, the questions addressed tend to 
raise additional issues and consequences. 

Initial concerns about students being able to function in the Realizeit platform were mostly 
unfounded. Students experienced relatively few issues and made a seamless transition to this new platform. 
In fact, from their survey responses, UCF’s concerns about preparing orientation and support materials may 
not have been necessary. At the underlying evaluation level, the latent trait analysis suggests that students 
evaluate their adaptive learning experience in much the same way that they do other course formats— 
design, delivery, learning facilitation, accurate assessment, understanding the rules of engagement and 
filtering out distraction and noise. 

Given students’ control over how, when, and for how long they approach learning, this modality 
appears to have been effective for a large majority—most of whom were successful. Many students 
completed the entirety of their assignments. The top performing students showed learning characteristics 
that were very similar to the class as a whole. A number of them took the opportunity to review and revise 
their work and some finished the course early. Others, however, had to be reminded of pending deadlines. 
In the second semester of the study the instructor opened the course and made it completely adaptive where 
students could address the modules in any order they preferred. Interestingly, however, the instructor 
reported the majority of students progressed through the material in the order in which it was described in 
the syllabus, essentially approaching their studies in a linear fashion. This leads the investigators to 
conclude that truly progressing to adaptive learning requires a period of adjustment and may take some 
time as well as a change in mindset. 

Although a small number of students in this project did not succeed in the course, the learning 
outcomes measures (module scores, knowledge covered and knowledge state) provided evidence of the 
difficulty potential for those students very early in the course. The slope of the line for those measures from 
Module 1 to Module 2 turned out to be an excellent predictor of success. The investigators speculate that 
this approach has an embedded analytics component and that measures external to the course may not be 
necessary to predict those at risk. This appears to be an important consideration since a major issue in 
analytics involves designing effective interventions. However, in Realizeit the interventions are relatively 
straightforward. Students need to complete the modules and revise their work until they have achieved 
competency and, if needed, be given additional needed time. 
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